{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 10.2 Keras介绍\n",
    "Keras是目前使用最为广泛的深度学习工具之一，它的底层可以支持TensorFlow、MXNet、CNTK和Theano。如今，Keras更是直接被引入TensorFlow的核心代码库，成为TensorFlow官方提供的高层封装之一。\n",
    "\n",
    "### 10.2.1 Keras基本用法\n",
    "**和TFLearn API类似，Keras API也对模型定义、损失定义、训练过程等进行了封装，而且封装之后的整个训练过程和TFLearn也基本一致，可以分为数据处理、模型定义、模型训练三个部分**。Keras安装：`pip install keras`。\n",
    "\n",
    "**1. Keras-CNN**\n",
    "\n",
    "以下代码展示了如何使用原生态Keras在MNIST数据集上实现LeNet-5模型："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 60000 samples, validate on 10000 samples\n",
      "Epoch 1/10\n",
      "60000/60000 [==============================] - 9s 152us/step - loss: 0.9914 - acc: 0.7421 - val_loss: 0.3020 - val_acc: 0.9108\n",
      "Epoch 2/10\n",
      "60000/60000 [==============================] - 3s 49us/step - loss: 0.2580 - acc: 0.9227 - val_loss: 0.1868 - val_acc: 0.9460\n",
      "Epoch 3/10\n",
      "60000/60000 [==============================] - 3s 49us/step - loss: 0.1852 - acc: 0.9446 - val_loss: 0.1452 - val_acc: 0.9583\n",
      "Epoch 4/10\n",
      "60000/60000 [==============================] - 3s 49us/step - loss: 0.1464 - acc: 0.9566 - val_loss: 0.1183 - val_acc: 0.9646\n",
      "Epoch 5/10\n",
      "60000/60000 [==============================] - 3s 49us/step - loss: 0.1222 - acc: 0.9637 - val_loss: 0.1052 - val_acc: 0.9692\n",
      "Epoch 6/10\n",
      "60000/60000 [==============================] - 3s 48us/step - loss: 0.1052 - acc: 0.9689 - val_loss: 0.0927 - val_acc: 0.9723\n",
      "Epoch 7/10\n",
      "60000/60000 [==============================] - 3s 48us/step - loss: 0.0940 - acc: 0.9724 - val_loss: 0.0834 - val_acc: 0.9746\n",
      "Epoch 8/10\n",
      "60000/60000 [==============================] - 3s 48us/step - loss: 0.0848 - acc: 0.9744 - val_loss: 0.0690 - val_acc: 0.9794\n",
      "Epoch 9/10\n",
      "60000/60000 [==============================] - 3s 49us/step - loss: 0.0774 - acc: 0.9768 - val_loss: 0.0704 - val_acc: 0.9782\n",
      "Epoch 10/10\n",
      "60000/60000 [==============================] - 3s 49us/step - loss: 0.0718 - acc: 0.9782 - val_loss: 0.0719 - val_acc: 0.9789\n",
      "10000/10000 [==============================] - 0s 50us/step\n",
      "Test loss: 0.07192779184058308\n",
      "Test accuracy: 0.9789\n"
     ]
    }
   ],
   "source": [
    "import keras\n",
    "from keras.datasets import mnist\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Flatten, Conv2D, MaxPooling2D\n",
    "from keras import backend as K\n",
    "\n",
    "# 1. 数据预处理\n",
    "num_classes = 10\n",
    "img_rows, img_cols = 28, 28\n",
    " \n",
    "# 通过Keras封装好的API加载MNIST数据。其中trainX就是一个60000 * 28 * 28的数组，\n",
    "# trainY是每一张图片对应的数字。\n",
    "(trainX, trainY), (testX, testY) = mnist.load_data()\n",
    "\n",
    "# 因为不同的底层（TensorFlow或MXNet）对输入的要求不一样，所以这里需要\n",
    "# 根据对图像编码的格式要求来设置输入层的格式。\n",
    "if K.image_data_format() == 'channels_first':\n",
    "    trainX = trainX.reshape(trainX.shape[0], 1, img_rows, img_cols)\n",
    "    testX = testX.reshape(testX.shape[0], 1, img_rows, img_cols)\n",
    "    input_shape = (1, img_rows, img_cols)\n",
    "else:\n",
    "    trainX = trainX.reshape(trainX.shape[0], img_rows, img_cols, 1)\n",
    "    testX = testX.reshape(testX.shape[0], img_rows, img_cols, 1)\n",
    "    input_shape = (img_rows, img_cols, 1)\n",
    "    \n",
    "# 将图像像素转化为0到1之间的实数\n",
    "trainX = trainX.astype('float32')\n",
    "testX = testX.astype('float32')\n",
    "trainX /= 255.0\n",
    "testX /= 255.0\n",
    " \n",
    "# 将标准答案转化为需要的格式（one-hot编码）。\n",
    "trainY = keras.utils.to_categorical(trainY, num_classes)\n",
    "testY = keras.utils.to_categorical(testY, num_classes)\n",
    "\n",
    "\n",
    "# 2. 使用Keras API定义模型。\n",
    "model = Sequential()\n",
    "\n",
    "# 一层深度为32，过滤器大小为5*5的卷积层\n",
    "model.add(Conv2D(32, kernel_size=(5, 5), activation='relu', input_shape=input_shape))\n",
    "# 一层过滤器大小为2*2的最大池化层\n",
    "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "# 一层深度为64，过滤器大小为5*5的卷积层\n",
    "model.add(Conv2D(64, (5, 5), activation='relu'))\n",
    "# 一层过滤器大小为2*2的最大池化层\n",
    "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "# 将卷积层的输出拉直后作为下面全连接层的输入\n",
    "model.add(Flatten())\n",
    "# 全连接层，由500个节点\n",
    "model.add(Dense(500, activation='relu'))\n",
    "# 全连接层，得到最后的输出\n",
    "model.add(Dense(num_classes, activation='softmax'))\n",
    " \n",
    "# 定义损失函数、优化函数和评测方法。\n",
    "model.compile(loss=keras.losses.categorical_crossentropy,\n",
    "              optimizer=keras.optimizers.SGD(),\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "\n",
    "# 3. 通过Keras的API训练模型并计算在测试数据上的准确率\n",
    "model.fit(trainX, trainY,\n",
    "          batch_size=128,\n",
    "          epochs=10,\n",
    "          validation_data=(testX, testY))\n",
    " \n",
    "# 在测试数据上计算准确率。\n",
    "score = model.evaluate(testX, testY)\n",
    "print('Test loss:', score[0])\n",
    "print('Test accuracy:', score[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "从以上代码可以看出使用Keras API训练模型过程为：\n",
    "1. **先定义一个Sequential类**；\n",
    "2. **然后在Sequential实例中通过add函数添加网络层**，Keras把卷积层、池化层、RNN结构（LSTM、GRN）、全连接层等常用的神经网络结构都做了封装，可以很方便地实现深层神经网络；\n",
    "3. **再通过compile函数指定优化函数、损失函数以及训练过程中需要监控的指标等**，Keras对优化函数、损失函数以及监控指标都有封装，同时也支持使用自定义的方式，在[Keras的API文档](https://keras.io/)中有详细的介绍，这里不再赘述；\n",
    "4. **最后Sequential实例可以通过fit函数来训练模型**，类似TFLearn中的fit函数， Keras的fit函数只须给出训练数据、batch大小和训练轮数，Keras就可以自动完成模型训练的整个过程。\n",
    "\n",
    "**2. Keras-RNN**\n",
    "\n",
    "除了能够很方便地处理图像问题，Keras对于循环神经网络的支持也是非常出色的。有了Keras API，循环神经网络的循环体结构也可以通过简单的一句命令完成。\n",
    "\n",
    "使用循环神经网络判断语言的情感（比如在以下例子中需要判断的一个评价是好评还是差评）和自然语言建模问题类似，唯一的区别在于除了最后一个时间点的输出是有意义的，其他时间点的输出都可以忽略。图10.1展示了使用循环神经网络处理情感分析问题的模型结构。\n",
    "<p align='center'>\n",
    "    <img src=images/图10.1.JPG>\n",
    "</p>\n",
    "\n",
    "以下代码给出了如何通过Keras实现自然语言情感分类问题。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://s3.amazonaws.com/text-datasets/imdb.npz\n",
      "17465344/17464789 [==============================] - 66s 4us/step\n",
      "25000 train sequences\n",
      "25000 test sequences\n",
      "trainX shape: (25000, 80)\n",
      "testX shape: (25000, 80)\n",
      "Train on 25000 samples, validate on 25000 samples\n",
      "Epoch 1/10\n",
      "25000/25000 [==============================] - 127s 5ms/step - loss: 0.4603 - acc: 0.7836 - val_loss: 0.3869 - val_acc: 0.8271\n",
      "Epoch 2/10\n",
      "25000/25000 [==============================] - 126s 5ms/step - loss: 0.3004 - acc: 0.8767 - val_loss: 0.3889 - val_acc: 0.8284\n",
      "Epoch 3/10\n",
      "25000/25000 [==============================] - 125s 5ms/step - loss: 0.2173 - acc: 0.9156 - val_loss: 0.4181 - val_acc: 0.8278\n",
      "Epoch 4/10\n",
      "25000/25000 [==============================] - 124s 5ms/step - loss: 0.1556 - acc: 0.9422 - val_loss: 0.4546 - val_acc: 0.8213\n",
      "Epoch 5/10\n",
      "25000/25000 [==============================] - 122s 5ms/step - loss: 0.1096 - acc: 0.9600 - val_loss: 0.5533 - val_acc: 0.8236\n",
      "Epoch 6/10\n",
      "25000/25000 [==============================] - 122s 5ms/step - loss: 0.0813 - acc: 0.9707 - val_loss: 0.6631 - val_acc: 0.8093\n",
      "Epoch 7/10\n",
      "25000/25000 [==============================] - 126s 5ms/step - loss: 0.0609 - acc: 0.9799 - val_loss: 0.7127 - val_acc: 0.8181\n",
      "Epoch 8/10\n",
      "25000/25000 [==============================] - 122s 5ms/step - loss: 0.0435 - acc: 0.9857 - val_loss: 0.9621 - val_acc: 0.8092\n",
      "Epoch 9/10\n",
      "25000/25000 [==============================] - 126s 5ms/step - loss: 0.0295 - acc: 0.9906 - val_loss: 0.8703 - val_acc: 0.8157\n",
      "Epoch 10/10\n",
      "25000/25000 [==============================] - 124s 5ms/step - loss: 0.0275 - acc: 0.9911 - val_loss: 0.9392 - val_acc: 0.8134\n",
      "25000/25000 [==============================] - 23s 902us/step\n",
      "Test loss: 0.9391764828938246\n",
      "Test accuracy: 0.8134\n"
     ]
    }
   ],
   "source": [
    "from keras.preprocessing import sequence\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Embedding\n",
    "from keras.layers import LSTM\n",
    "from keras.datasets import imdb\n",
    "\n",
    "# 1. 数据预处理\n",
    "max_features = 20000     # 最多使用的单词数\n",
    "maxlen = 80              # 循环网络的截断长度\n",
    "batch_size = 32\n",
    "\n",
    "# 加载数据并将单词转化为ID。和自然语言模型类似，会将出现频率较低的单词替换为统一的ID。\n",
    "# 通过keras封装的API会生成25000条训练数据和25000条测试数据，\n",
    "# 每一条数据可以被看成一段话，并且每段话都有一个好评或者差评的标签。\n",
    "(trainX, trainY), (testX, testY) = imdb.load_data(num_words=max_features)\n",
    "print(len(trainX), 'train sequences')\n",
    "print(len(testX), 'test sequences')\n",
    "\n",
    "# 在自然语言中，每一段话的长度是不一样的，但循环神经网络的循环长度是固定的，\n",
    "# 所以这里需要先将所有段落统一成固定长度。不够的填充0，超过的忽略超多的部分。\n",
    "trainX = sequence.pad_sequences(trainX, maxlen=maxlen)\n",
    "testX = sequence.pad_sequences(testX, maxlen=maxlen)\n",
    "print('trainX shape:', trainX.shape)\n",
    "print('testX shape:', testX.shape)\n",
    "\n",
    "\n",
    "# 2. 模型定义\n",
    "model = Sequential()\n",
    "\n",
    "# 构建Embedding层，128代表了embedding向量的维度。\n",
    "model.add(Embedding(max_features, 128))\n",
    "# 构建LSTM层。\n",
    "model.add(LSTM(128, dropout=0.2, recurrent_dropout=0.2))\n",
    "# 构建最后的全连接层，注意上面在构建的LSTM层时只会得到最后一个节点的输出。\n",
    "# 如果需要输出每个节点的结果，可以将return_sequence设置为True\n",
    "model.add(Dense(1, activation='sigmoid'))\n",
    "\n",
    "# 与MNIST样例类似地指定损失函数、优化方法和评估指标。\n",
    "model.compile(loss='binary_crossentropy',\n",
    "              optimizer='adam',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "\n",
    "# 3. 模型训练、测试\n",
    "# 与MNIST样例类似地指定训练数据、训练轮数、batch大小以及验证数据。\n",
    "model.fit(trainX, trainY,\n",
    "          batch_size=batch_size,\n",
    "          epochs=10,\n",
    "          validation_data=(testX, testY))\n",
    "\n",
    "score = model.evaluate(testX, testY, batch_size=batch_size)\n",
    "print('Test loss:', score[0])\n",
    "print('Test accuracy:', score[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 10.2.2 keras高级用法\n",
    "10.2.1小节对Keras基本用法进行了详细介绍，虽然通过Keras地封装，很多经典地神经网络结构可以很快地被实现，不过要实现一些更加灵活地网络结构、损失函数或者数据输入方法，就需要对Keras地高级用法有更多的了解。\n",
    "\n",
    "**1. 返回值形式定义网络（函数式）**\n",
    "\n",
    "上小节中一个最重要地封装就是Sequential类，所有神经网络模型定义和训练都是通过这个类实现的。然而从其名字就可以看出，**Sequential只支持顺序模型地定义**。类似Inception这样的模型结构，通过Sequential类就不容易直接实现了，**为了支持更加灵活的模型定义方法，Keras支持以返回值的形式定义网络层结构**。可以参考[Keras官网文档对这两种构建网络的介绍](https://keras.io/models/about-keras-models/)。下面代码展示了如何使用这种方式定义模型："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 60000 samples, validate on 10000 samples\n",
      "Epoch 1/10\n",
      "60000/60000 [==============================] - 9s 150us/step - loss: 0.6081 - acc: 0.8498 - val_loss: 0.3436 - val_acc: 0.9068\n",
      "Epoch 2/10\n",
      "60000/60000 [==============================] - 5s 78us/step - loss: 0.3235 - acc: 0.9107 - val_loss: 0.2790 - val_acc: 0.9220\n",
      "Epoch 3/10\n",
      "60000/60000 [==============================] - 5s 78us/step - loss: 0.2751 - acc: 0.9229 - val_loss: 0.2472 - val_acc: 0.9318\n",
      "Epoch 4/10\n",
      "60000/60000 [==============================] - 5s 78us/step - loss: 0.2444 - acc: 0.9325 - val_loss: 0.2230 - val_acc: 0.9386\n",
      "Epoch 5/10\n",
      "60000/60000 [==============================] - 5s 78us/step - loss: 0.2212 - acc: 0.9389 - val_loss: 0.2074 - val_acc: 0.9419\n",
      "Epoch 6/10\n",
      "60000/60000 [==============================] - 5s 80us/step - loss: 0.2024 - acc: 0.9441 - val_loss: 0.1913 - val_acc: 0.9470\n",
      "Epoch 7/10\n",
      "60000/60000 [==============================] - 5s 81us/step - loss: 0.1868 - acc: 0.9480 - val_loss: 0.1797 - val_acc: 0.9486\n",
      "Epoch 8/10\n",
      "60000/60000 [==============================] - 5s 80us/step - loss: 0.1735 - acc: 0.9519 - val_loss: 0.1676 - val_acc: 0.9523\n",
      "Epoch 9/10\n",
      "60000/60000 [==============================] - 5s 81us/step - loss: 0.1617 - acc: 0.9555 - val_loss: 0.1581 - val_acc: 0.9534\n",
      "Epoch 10/10\n",
      "60000/60000 [==============================] - 5s 82us/step - loss: 0.1517 - acc: 0.9586 - val_loss: 0.1509 - val_acc: 0.9560\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x2cca27d0d30>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 对比第一个的cell\n",
    "\n",
    "import keras\n",
    "from tflearn.layers.core import fully_connected\n",
    "from keras.datasets import mnist\n",
    "from keras.layers import Input, Dense\n",
    "from keras.models import Model\n",
    "\n",
    "# 1. 数据预处理\n",
    "num_classes = 10\n",
    "img_rows, img_cols = 28, 28\n",
    " \n",
    "# 通过Keras封装好的API加载MNIST数据。\n",
    "# 对比10.2.1中keras-CNN，由于这里使用全连接层，不需要将输入整理成三维矩阵\n",
    "(trainX, trainY), (testX, testY) = mnist.load_data()\n",
    "trainX = trainX.reshape(trainX.shape[0], img_rows * img_cols)\n",
    "testX = testX.reshape(testX.shape[0], img_rows * img_cols)\n",
    " \n",
    "# 将图像像素转化为0到1之间的实数。\n",
    "trainX = trainX.astype('float32')\n",
    "testX = testX.astype('float32')\n",
    "trainX /= 255.0\n",
    "testX /= 255.0\n",
    " \n",
    "# 将标准答案转化为需要的格式（one-hot编码）。\n",
    "trainY = keras.utils.to_categorical(trainY, num_classes)\n",
    "testY = keras.utils.to_categorical(testY, num_classes)\n",
    "\n",
    "\n",
    "# 2. 通过返回值的方式定义模型\n",
    "inputs = Input(shape=(784,))\n",
    "\n",
    "x = Dense(500, activation='relu')(inputs)\n",
    "predictions = Dense(10, activation='softmax')(x)\n",
    "\n",
    "model = Model(inputs=inputs, outputs=predictions)\n",
    "model.compile(loss=keras.losses.categorical_crossentropy,\n",
    "              optimizer=keras.optimizers.SGD(),\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "\n",
    "# 3. 训练模型\n",
    "model.fit(trainX, trainY,\n",
    "          batch_size=32,\n",
    "          epochs=10,\n",
    "          validation_data=(testX, testY))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "通过这样的方式，Keras就可以实现类似Inception这样的模型结构，以下代码展示了如何通过Keras实现Inception结构："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "from keras.layers import Conv2D, MaxPooling2D, Input\n",
    "\n",
    "# 定义输入图像的尺寸\n",
    "input_img = Input(shape=(256, 256, 3))\n",
    "\n",
    "# 定义第一个分支\n",
    "tower_1 = Conv2D(64, (1, 1), padding='same', activation='relu')(input_img)\n",
    "tower_1 = Conv2D(64, (3, 3), padding='same', activation='relu')(tower_1)\n",
    "\n",
    "# 定义第二个分支。与顺序模型不同，第二个分支的输入时input_img，而不是第一分支的输出\n",
    "tower_2 = Conv2D(64, (1, 1), padding='same', activation='relu')(input_img)\n",
    "tower_2 = Conv2D(64, (5, 5), padding='same', activation='relu')(tower_2)\n",
    "\n",
    "# 定义第三个分支。类似的，输入也是input_img\n",
    "tower_3 = MaxPooling2D((3, 3), strides=(1, 1), padding='same')(input_img)\n",
    "tower_3 = Conv2D(64, (1, 1), padding='same', activation='relu')(tower_3)\n",
    "\n",
    "# 将三个分支通过concatenate的方式拼接在一起\n",
    "output = keras.layers.concatenate([tower_1, tower_2, tower_3], axis=1)\n",
    "'''"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**2. 多输入输出**\n",
    "\n",
    "除了可以支持非顺序模型，Keras也可以支持有多个输入或输出的模型，下图是一种多输入输出的网络结构：\n",
    "<p align='center'>\n",
    "    <img src=images/图10.2.JPG>\n",
    "</p>\n",
    "\n",
    "- 输入层1含有784个节点，代表MNIST图片中的784个像素；\n",
    "- 输入层2含有10个节点，代表该图片所对应的数字；\n",
    "- 输出层1在预测时仅依赖维度为1的隐藏层，因此预测的准确度较低；\n",
    "- 输出层2中直接包含了正确答案，因此准确率较高。\n",
    "\n",
    "下面代码给出了如何实现上图的网络结构："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 60000 samples, validate on 10000 samples\n",
      "Epoch 1/20\n",
      "60000/60000 [==============================] - 6s 102us/step - loss: 2.4584 - output1_loss: 2.2186 - output2_loss: 2.3984 - output1_acc: 0.1795 - output2_acc: 0.1022 - val_loss: 2.4028 - val_output1_loss: 2.1664 - val_output2_loss: 2.3639 - val_output1_acc: 0.1963 - val_output2_acc: 0.1010\n",
      "Epoch 2/20\n",
      "60000/60000 [==============================] - 2s 30us/step - loss: 2.3731 - output1_loss: 2.1404 - output2_loss: 2.3272 - output1_acc: 0.1973 - output2_acc: 0.1022 - val_loss: 2.3367 - val_output1_loss: 2.1075 - val_output2_loss: 2.2922 - val_output1_acc: 0.1944 - val_output2_acc: 0.1010\n",
      "Epoch 3/20\n",
      "60000/60000 [==============================] - 2s 29us/step - loss: 2.3146 - output1_loss: 2.0900 - output2_loss: 2.2459 - output1_acc: 0.2088 - output2_acc: 0.1719 - val_loss: 2.2830 - val_output1_loss: 2.0628 - val_output2_loss: 2.2018 - val_output1_acc: 0.2272 - val_output2_acc: 0.1932\n",
      "Epoch 4/20\n",
      "60000/60000 [==============================] - 2s 29us/step - loss: 2.2608 - output1_loss: 2.0453 - output2_loss: 2.1548 - output1_acc: 0.2286 - output2_acc: 0.2083 - val_loss: 2.2288 - val_output1_loss: 2.0179 - val_output2_loss: 2.1081 - val_output1_acc: 0.2356 - val_output2_acc: 0.2128\n",
      "Epoch 5/20\n",
      "60000/60000 [==============================] - 2s 29us/step - loss: 2.2048 - output1_loss: 1.9982 - output2_loss: 2.0659 - output1_acc: 0.2303 - output2_acc: 0.2586 - val_loss: 2.1725 - val_output1_loss: 1.9705 - val_output2_loss: 2.0203 - val_output1_acc: 0.2346 - val_output2_acc: 0.3125\n",
      "Epoch 6/20\n",
      "60000/60000 [==============================] - 2s 29us/step - loss: 2.1510 - output1_loss: 1.9523 - output2_loss: 1.9876 - output1_acc: 0.2341 - output2_acc: 0.3250 - val_loss: 2.1232 - val_output1_loss: 1.9284 - val_output2_loss: 1.9485 - val_output1_acc: 0.2456 - val_output2_acc: 0.3411\n",
      "Epoch 7/20\n",
      "60000/60000 [==============================] - 2s 29us/step - loss: 2.1067 - output1_loss: 1.9144 - output2_loss: 1.9225 - output1_acc: 0.2411 - output2_acc: 0.3419 - val_loss: 2.0839 - val_output1_loss: 1.8952 - val_output2_loss: 1.8872 - val_output1_acc: 0.2450 - val_output2_acc: 0.3493\n",
      "Epoch 8/20\n",
      "60000/60000 [==============================] - 2s 29us/step - loss: 2.0718 - output1_loss: 1.8853 - output2_loss: 1.8655 - output1_acc: 0.2430 - output2_acc: 0.3545 - val_loss: 2.0534 - val_output1_loss: 1.8701 - val_output2_loss: 1.8327 - val_output1_acc: 0.2436 - val_output2_acc: 0.3672\n",
      "Epoch 9/20\n",
      "60000/60000 [==============================] - 2s 29us/step - loss: 2.0440 - output1_loss: 1.8626 - output2_loss: 1.8134 - output1_acc: 0.2459 - output2_acc: 0.3788 - val_loss: 2.0289 - val_output1_loss: 1.8506 - val_output2_loss: 1.7833 - val_output1_acc: 0.2488 - val_output2_acc: 0.3962\n",
      "Epoch 10/20\n",
      "60000/60000 [==============================] - 2s 29us/step - loss: 2.0210 - output1_loss: 1.8445 - output2_loss: 1.7651 - output1_acc: 0.2499 - output2_acc: 0.4154 - val_loss: 2.0084 - val_output1_loss: 1.8347 - val_output2_loss: 1.7365 - val_output1_acc: 0.2532 - val_output2_acc: 0.4401\n",
      "Epoch 11/20\n",
      "60000/60000 [==============================] - 2s 29us/step - loss: 2.0019 - output1_loss: 1.8299 - output2_loss: 1.7194 - output1_acc: 0.2534 - output2_acc: 0.4560 - val_loss: 1.9905 - val_output1_loss: 1.8213 - val_output2_loss: 1.6921 - val_output1_acc: 0.2584 - val_output2_acc: 0.4796\n",
      "Epoch 12/20\n",
      "60000/60000 [==============================] - 2s 29us/step - loss: 1.9854 - output1_loss: 1.8178 - output2_loss: 1.6759 - output1_acc: 0.2563 - output2_acc: 0.5143 - val_loss: 1.9746 - val_output1_loss: 1.8097 - val_output2_loss: 1.6493 - val_output1_acc: 0.2628 - val_output2_acc: 0.6113\n",
      "Epoch 13/20\n",
      "60000/60000 [==============================] - 2s 29us/step - loss: 1.9709 - output1_loss: 1.8075 - output2_loss: 1.6338 - output1_acc: 0.2602 - output2_acc: 0.7036 - val_loss: 1.9616 - val_output1_loss: 1.8008 - val_output2_loss: 1.6088 - val_output1_acc: 0.2527 - val_output2_acc: 0.7073\n",
      "Epoch 14/20\n",
      "60000/60000 [==============================] - 2s 30us/step - loss: 1.9580 - output1_loss: 1.7987 - output2_loss: 1.5934 - output1_acc: 0.2617 - output2_acc: 0.7177 - val_loss: 1.9488 - val_output1_loss: 1.7920 - val_output2_loss: 1.5685 - val_output1_acc: 0.2674 - val_output2_acc: 0.7255\n",
      "Epoch 15/20\n",
      "60000/60000 [==============================] - 2s 29us/step - loss: 1.9464 - output1_loss: 1.7910 - output2_loss: 1.5541 - output1_acc: 0.2664 - output2_acc: 0.7298 - val_loss: 1.9378 - val_output1_loss: 1.7848 - val_output2_loss: 1.5303 - val_output1_acc: 0.2684 - val_output2_acc: 0.7355\n",
      "Epoch 16/20\n",
      "60000/60000 [==============================] - 2s 29us/step - loss: 1.9359 - output1_loss: 1.7842 - output2_loss: 1.5162 - output1_acc: 0.2658 - output2_acc: 0.7392 - val_loss: 1.9277 - val_output1_loss: 1.7783 - val_output2_loss: 1.4935 - val_output1_acc: 0.2718 - val_output2_acc: 0.7452\n",
      "Epoch 17/20\n",
      "60000/60000 [==============================] - 2s 29us/step - loss: 1.9260 - output1_loss: 1.7781 - output2_loss: 1.4795 - output1_acc: 0.2704 - output2_acc: 0.7907 - val_loss: 1.9182 - val_output1_loss: 1.7725 - val_output2_loss: 1.4572 - val_output1_acc: 0.2837 - val_output2_acc: 0.8414\n",
      "Epoch 18/20\n",
      "60000/60000 [==============================] - 2s 33us/step - loss: 1.9169 - output1_loss: 1.7726 - output2_loss: 1.4435 - output1_acc: 0.2726 - output2_acc: 0.8457 - val_loss: 1.9096 - val_output1_loss: 1.7674 - val_output2_loss: 1.4224 - val_output1_acc: 0.2846 - val_output2_acc: 0.8471\n",
      "Epoch 19/20\n",
      "60000/60000 [==============================] - 2s 31us/step - loss: 1.9085 - output1_loss: 1.7677 - output2_loss: 1.4089 - output1_acc: 0.2764 - output2_acc: 0.8513 - val_loss: 1.9014 - val_output1_loss: 1.7627 - val_output2_loss: 1.3876 - val_output1_acc: 0.2794 - val_output2_acc: 0.8544\n",
      "Epoch 20/20\n",
      "60000/60000 [==============================] - 2s 31us/step - loss: 1.9006 - output1_loss: 1.7632 - output2_loss: 1.3750 - output1_acc: 0.2766 - output2_acc: 0.8634 - val_loss: 1.8935 - val_output1_loss: 1.7580 - val_output2_loss: 1.3543 - val_output1_acc: 0.2892 - val_output2_acc: 0.8720\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x2ce033c1cf8>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import keras\n",
    "from tflearn.layers.core import fully_connected\n",
    "from keras.datasets import mnist\n",
    "from keras.layers import Input, Dense\n",
    "from keras.models import Model\n",
    "\n",
    "# 1. 数据预处理\n",
    "num_classes = 10\n",
    "img_rows, img_cols = 28, 28\n",
    " \n",
    "# 通过Keras封装好的API加载MNIST数据。\n",
    "(trainX, trainY), (testX, testY) = mnist.load_data()\n",
    "trainX = trainX.reshape(trainX.shape[0], img_rows * img_cols)\n",
    "testX = testX.reshape(testX.shape[0], img_rows * img_cols)\n",
    "\n",
    "trainX = trainX.astype('float32')\n",
    "testX = testX.astype('float32')\n",
    "trainX /= 255.0\n",
    "testX /= 255.0\n",
    "\n",
    "trainY = keras.utils.to_categorical(trainY, num_classes)\n",
    "testY = keras.utils.to_categorical(testY, num_classes)\n",
    "\n",
    "\n",
    "# 2. 定义模型\n",
    "# 定义两个输入。\n",
    "input1 = Input(shape=(784,), name = \"input1\")\n",
    "input2 = Input(shape=(10,), name = \"input2\")\n",
    "\n",
    "# 定义第一个输出。\n",
    "x = Dense(1, activation='relu')(input1)\n",
    "output1 = Dense(10, activation='softmax', name = \"output1\")(x)\n",
    "\n",
    "# 定义第二个输出。\n",
    "y = keras.layers.concatenate([x, input2])\n",
    "output2 = Dense(10, activation='softmax', name = \"output2\")(y)\n",
    "\n",
    "# 定义一个有多个输入和输出的模型，这里需要将所有的输入和输出给出即可\n",
    "model = Model(inputs=[input1, input2], outputs=[output1, output2])\n",
    "\n",
    "# 定义损失函数、优化函数和评测方法。\n",
    "model.compile(loss=keras.losses.categorical_crossentropy,\n",
    "              optimizer=keras.optimizers.SGD(),\n",
    "              loss_weights = [1, 0.1],\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "\n",
    "# 3. 训练模型\n",
    "model.fit([trainX, trainY], [trainY, trainY],\n",
    "          batch_size=128,\n",
    "          epochs=20,\n",
    "          validation_data=([testX, testY], [testY, testY]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对于多输入输出需要注意的是：\n",
    "- **对于损失函数**：若多个输出的损失函数相间，可以只指定一个损失函数；若多个输出的损失函数不同，则可以通过一个列表或一个字典来指定每一个输出的损失函数，比如可以使用：`loss={'output1':'binary_crossentropy', 'output2':'binary_crossentropy'｝`来为不同的输出指定不同的损失函数。\n",
    "\n",
    "- **不同损失权重**：类似地， Keras也支持为不同输出产生的损失指定权重，这可以通过loss_weights参数来完成。比如在上例的定义中，output1的权重为1,output2的权重为0.1，所以这个模型会更加偏向于优化第一个输出。\n",
    "\n",
    "- **训练的输入**：在模型训练过程，因为有两个输入和输出，所以这里提供的数据也需要有两个输入和两个期待的正确答案输出。通过列表的方式提供数据时， Keras会假设数据给出的顺序和定义Model类时输入输出给出的顺序是对应的。为了避免顺序不一致导致的问题，本书更推荐使用字典的形式给出：\n",
    "\n",
    "`model.fit(\n",
    "  {'input1': trainX,'input2': trainY},\n",
    "  {'output1': trainY,'output2': trainY},\n",
    "   ...)\n",
    "`\n",
    "\n",
    "从以上输出可以看出Keras在训练过程中会显示每个输出层的loss和accuracy。因为输出层output1只使用了一个维度为1的隐藏节点，所以正确率只有29.85%。虽然输出层output2使用了正确答案作为输入，但是因为在损失函数中权重较低（只有0.1），所以它的收敛速度较慢，在20个epoch 时准确率也只有86.3%。如果将两个输出层的损失权重设为一样，那么输出层output1在20个epoch时的准确率将只有27%，而输出层output2的准确率可以达到99.9%。\n",
    "\n",
    "**3. Keras + 原生态TensorFlow API**\n",
    "\n",
    "虽然通过返回值的方式已经可以实现大部分的神经网络模型，然而Keras API还存在两大问题：\n",
    "- 第一，原生态Keras API对训练数据的处理流程支持得不太好，基本上需要一次性将数据全部加载到内存；\n",
    "- 第二，原生态Keras API无法支持分布式训练。\n",
    "\n",
    "为了解决这两个问题，Keras提供了一种与原生态TensorFlow结合得更加紧密的方式。以下代码显示了如何联合二者来解决MNIST问题："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From <ipython-input-1-1a94e15ac683>:4: read_data_sets (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use alternatives such as official/mnist/dataset.py from tensorflow/models.\n",
      "WARNING:tensorflow:From d:\\python3\\tfgpu\\dl+\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\datasets\\mnist.py:260: maybe_download (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please write your own downloading logic.\n",
      "WARNING:tensorflow:From d:\\python3\\tfgpu\\dl+\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\datasets\\mnist.py:262: extract_images (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use tf.data to implement this functionality.\n",
      "Extracting ../../datasets/MNIST_data\\train-images-idx3-ubyte.gz\n",
      "WARNING:tensorflow:From d:\\python3\\tfgpu\\dl+\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\datasets\\mnist.py:267: extract_labels (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use tf.data to implement this functionality.\n",
      "Extracting ../../datasets/MNIST_data\\train-labels-idx1-ubyte.gz\n",
      "WARNING:tensorflow:From d:\\python3\\tfgpu\\dl+\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\datasets\\mnist.py:110: dense_to_one_hot (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use tf.one_hot on tensors.\n",
      "Extracting ../../datasets/MNIST_data\\t10k-images-idx3-ubyte.gz\n",
      "Extracting ../../datasets/MNIST_data\\t10k-labels-idx1-ubyte.gz\n",
      "WARNING:tensorflow:From d:\\python3\\tfgpu\\dl+\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\datasets\\mnist.py:290: DataSet.__init__ (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use alternatives such as official/mnist/dataset.py from tensorflow/models.\n",
      "After 0 training step(s), loss on training batch is 2.37641.\n",
      "After 1000 training step(s), loss on training batch is 0.0315625.\n",
      "After 2000 training step(s), loss on training batch is 0.0420956.\n",
      "After 3000 training step(s), loss on training batch is 0.00413271.\n",
      "After 4000 training step(s), loss on training batch is 0.00997246.\n",
      "After 5000 training step(s), loss on training batch is 0.00410553.\n",
      "After 6000 training step(s), loss on training batch is 0.00987568.\n",
      "After 7000 training step(s), loss on training batch is 0.00476654.\n",
      "After 8000 training step(s), loss on training batch is 0.00376713.\n",
      "After 9000 training step(s), loss on training batch is 0.00289015.\n",
      "Final test accuracy is 0.983200.\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "\n",
    "mnist_data = input_data.read_data_sets('../../datasets/MNIST_data', one_hot=True)\n",
    "\n",
    "# 通过TensorFlow中的placeholder定义输入。类似地，Keras封装的网络层结构也可以支持\n",
    "# 使用第7章中介绍的输入队列。这样可以有效避免一次性加载所有数据的问题。\n",
    "x = tf.placeholder(tf.float32, shape=(None, 784))\n",
    "y_ = tf.placeholder(tf.float32, shape=(None, 10))\n",
    "\n",
    "# 直接使用TensorFlow中提供的Keras API定义网络层结构\n",
    "net = tf.keras.layers.Dense(500, activation='relu')(x)\n",
    "y = tf.keras.layers.Dense(10, activation='softmax')(net)\n",
    "\n",
    "# 定义损失函数和优化方法。注意这里可以混用Keras的API和原生态TensorFlow的API\n",
    "loss = tf.reduce_mean(tf.keras.losses.categorical_crossentropy(y_, y))\n",
    "train_step = tf.train.GradientDescentOptimizer(0.5).minimize(loss)\n",
    "\n",
    "# 定义预测正确率作为指标\n",
    "acc_value = tf.reduce_mean(tf.keras.metrics.categorical_accuracy(y_, y))\n",
    "\n",
    "# 使用原生态TensorFlow的方式训练模型，这样可以有效实现分布式。\n",
    "with tf.Session() as sess:\n",
    "    tf.global_variables_initializer().run()\n",
    "    for i in range(10000):\n",
    "        xs, ys = mnist_data.train.next_batch(100)\n",
    "        _, loss_value = sess.run([train_step, loss], feed_dict={x: xs, y_: ys})\n",
    "        if i % 1000 == 0:\n",
    "            print(\"After %d training step(s), loss on training batch is \"\n",
    "                  \"%g.\" % (i, loss_value))\n",
    "\n",
    "    test_acc = acc_value.eval(feed_dict={x: mnist_data.test.images,\n",
    "                                    y_: mnist_data.test.labels})\n",
    "    print('Final test accuracy is %f.' % test_acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "通过Keras和原生态TensorFlow更紧密地结合，可以使建模的灵活性进一步提高，但是同时也会损失一部分封装带来的易用性。所以在实际问题中读者可以根据需求合理地选择封装的程度。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
